Learning to Evaluate Performance of Multimodal Semantic Localization

Semantic localization (SeLo) refers to the task of obtaining most relevant locations in large-scale remote sensing (RS) images using semantic information such as text. As an emerging based on cross-modal retrieval, SeLo achieves semantic-level retrieval with only caption-level annotation, which demonstrates its great potential unifying downstream tasks. Although has been carried out successively, but there is currently no work systematically explores and analyzes this urgent direction. In paper, we thoroughly study field provide a complete benchmark terms metrics testdata advance task. Firstly, characteristics task, propose multiple discriminative evaluation quantify performance The devised significant area proportion, attention shift distance, discrete distance are utilized evaluate generated map from pixel-level region-level. Next, standard data for contribute diverse, multi-semantic, multi-objective Localization Testset (AIR-SLT). AIR-SLT consists 22 RS 59 test cases different semantics, aims comprehensive evaluations models. Finally, analyze models detail, explore impact variables We have also established new paradigm referring expression comprehension, demonstrated advantage semantics through combining it tasks detection road extraction. proposed metrics, testsets, corresponding scripts open access at https://github.com/xiaoyuan1996/SemanticLocalizationMetrics.